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SYNCHRONIZING SERVICES ACROSS OPEN CABLE AND PACKET CABLE 

Background of the Invention 

a. Field of the Invention 

The present invention pertains generally to digital data transferred over a network 
in conjunction with video or audio data and more specifically to synchronization of data 
transferred over a network with video or audio data transferred on that network or on an 
independent network or channel. 

b. Description of the Background 

Television and audio broadcast may employ a range of broadband media 
including cable, satellite and high speed Internet. The term broadband refers to the 
capability of the media or network to carry multiple channels at one time. Video and 
audio programs are typically allocated a constant bandwidth such that dropouts or 
distorted output is not produced. A channel that provides a constant minimum bandwidth 
rate may be referred to as isochronous or as a constant bit rate (CBR) channel. Other 
services that transfer data on the network, such as Internet, have typically been 
asynchronous, wherein bandwidth may vary depending on network load. New standards 
are emerging that provide for isochronous transfer of data on a network. Such standards 
include voice over DSL (Digital Subscriber Line) and DOCSIS (Data Over Cable System 
Interface Specification) and more specifically by PacketCable that provides a method for 
isochronous data transfers over cable. 

The advent of isochronous data transfer over broadband networks enables 
streaming audio and video programs other than those provided by a cable or television 
operator. Additionally, isochronous data transfers allow the possibility for data from one 
channel to be synchronized with an audio/video broadcast stream or with data from 
another channel. Synchronized data transfer may be employed to provide modification 
and enhancement of a broadcast program. For example, subtitles in a number of spoken 
languages, or sign language, may be added to televised programs. Such additions are 
outside the capabilities of captioning facilities of televisions. Additionally, synchronized 
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data transfers maybe used to add local content to advertising, such as local automobile 
dealer information presented in conjunction with a national automobile advertisement. 
Data transfers must be synchronized in some manner such that subtitles appear at the 
correct time or advertising message enhancements appear at the correct time. 
5 Synchronized data transfers utilized in conjunction with an audio/video stream may 
provide increased viewer enjoyment and may provide additional opportunities for 
revenue. As such, a new method is needed to control the transfer of data and the 
utilization of that data in synchrony with an audio/video stream or data from another 
channel. 

10 

Summary of the Invention 

The present invention overcomes the disadvantages and limitations of the prior art 
by providing a system and method wherein data transferred over an isochronous channel 

1 5 may be synchronized with an audio or video stream. Time information contained within 
an audio or video stream, such as an MPEG audio/video stream, is employed to 
synchronize the utilization of data from the isochronous channel. MPEG formats are 
employed in cable, satellite, DVD and other systems. Information describing the stream, 
time and nature of combination with a data transfer may be contained within the 

20 audio/video stream, the data transfer, or may be supplied by other methods. 

The present invention may therefore comprise a method for synchronizing data 
transfer with an audio/video stream comprising: receiving an audio/video stream; 
receiving a data transfer including a descriptor of when the data transfer may be 
combined with the audio/video stream; obtaining time information from the audio/video 

25 stream; and combining the data transfer and the audio/video stream when the time 
information is in substantial agreement with the descriptor. 

The invention may further comprise a system for combining a data transfer with 
an audio/video stream comprising: a video server; a data server; a client system operable 
to receive the audio/video stream from the video server and the data transfer from the 

30 data server; a network management component operable to provide isochronous data 
transfer from the data server to the client system; and a processing component in the 
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client system operable to monitor the audio/video stream for time information and to 
combine the data transfer with the audio/video stream. 

Latencies incurred between requests for data and reception of data over the 
channel may be measured and the time at which data is requested may be adjusted in 
5 order to minimize local memory requirements. Advantages of the present invention 
include the ability to overlay images from a data transfer with an audio/video stream. 
Such overlays maybe used for advertising, entertainment or education. Advertising may 
employ an overlay to identify products for sale in a video image, or to display a product 
ordering menu or product information. An entertainment application may employ the 

10 present invention to combine an audio/video stream, used for background images, with a 
data transfer used to overlay game characters or other images. Further, the invention may 
be employed to display a musical score or other information such as guitar chords when a 
musical audio/video stream is presented. In addition, the present invention allows 
personalization of advertising at the customer level whereas other personalization 

15 methods are often implemented at the broadcast site, limiting the number of versions that 
may be produced. In other words, the present invention allows modification of an 
audio/video stream by the client system in contrast to methods where modification is 
performed by a server at the head end of a cable or satellite system and is therefore 
limited by the number of broadcast channels. 

20 Further yet, the synchronization of data transfers and an audio/video stream may 

be employed to reduce processing requirements of a client system for television based 
advertising and purchasing. A client system may be a set top box, satellite receiver or 
personal computer, or other system capable of networking and audio/video output. In 
contrast to executing menu software on a client system, the system of the present 

25 invention may be employed to return audio/video stream identification, frame number, 
and mouse click position from the client system to a server. The server then may generate 
menus, product information, or other displays and transfer them to the client system for 
combination with an audio/video stream. 

Further advantages of the present invention include that an audio/video 

30 presentation need not be modified to include markers or triggers, as is the case for 

ATVEF (Advanced Television Enhancement Foundation). By using timing information 
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already contained in a broadcast or streaming presentation, the present invention allows 
greater flexibility in content enhancement. 

Brief Description of the Drawings 

In the drawings, 

FIGURE 1 is a system overview block diagram. 

FIGURE 2 depicts a flowchart for synchronizing downstream data transfers with 
an audio/video stream. 

FIGURE 3 depicts a flowchart for personalizing data transfers utilized with an 
audio/video stream. 

Detailed Description of the Invention 

The present invention provides a system and method for the transfer of data and 
use of the data in synchrony with an audio/video stream. Data transfers may be on a 
separate channel or network and may employ protocols as defined by DOCSIS (Data 
Over Cable System Interface Specification) versions including PacketCable or other 
standards such that the network or channel provides isochronous data transfer. 

An audio/video stream may be in MPEG-2 format, as may be used in satellite and 
digital cable television systems. MPEG-2 employs data packets of 1 88 bytes each. At the 
start of each packet is a package identifier (PID). There are four types of PIDs used by 
satellite receivers. A VPID is the PID for the video stream and an APID is the PID for an 
audio stream. A PGR PID (program clock reference) may be used to synchronize the 
video and audio packets. Clock reference data is often embedded into the video stream. 
The present invention may employ timing information from the PCR PID or the timing 
information contained within the video stream or audio stream to synchronize utilization 
of transferred data. A fourth PID is the System Information PID and may be used for data 
such as a program guide, and information about other frequencies. In one embodiment of 
the present invention, the System Information PID may be employed to indicate that data 
is available in conjunction with the present audio/video stream. 

Data transfers may be broadcast in conjunction with a program, or may be 
transferred in response to a request from a client system, such as a set top box, satellite 
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receiver, or personal computer. A software program in the client system controls the 
utilization of data transfers with an audio/video stream. When data transfers are broadcast 
in conjunction with an audio/video stream, the data transfers may contain information 
that identifies the audio/video stream with which the data may be utilized, and the time at 
5 which the data may be utilized. Further, the data transfer may contain information 
describing data type and how the data is to be used. For example, a broadcast data 
transfer containing names of local automobile dealers to be used in conjunction with an 
advertising message may include a descriptor that text data of a predetermined font size is 
to be overlaid at a specified position on the display screen. Further, information contained 
1 0 within the data transfer or audio/video stream may define the type of combination to be 
performed between the transferred data and the audio/video stream. The combination 
may employ a number of methods including logical operations, blending, or conditional 
combination, for example. Further, combination may include windowing wherein 
information from an audio/video stream may be placed in a window and information 
1 5 generated using a data transfer may be placed in another window or combination of 

windows. Combination may be performed using digital data in a display memory or may 
employ analog signal mixing. 

As previously mentioned, data transfers may be broadcast in conjunction with a 
program, or may be transferred in response to a request from a client system. When data 
20 transfers are performed in response to a request from a client system, the client system 
may include methods to estimate or measure the amount of time required to establish a 
connection and receive a data transfer. Advantageously, these methods may reduce the 
amount of memory needed in the client system to save data transfers prior to utilization. 
A software program in the client system may contain downloaded information describing 
25 which audio/video streams may have associated data transfers. The downloaded 

information may be received as part of a data transfer employing an isochronous data 
channel or maybe transferred by other methods such as an asynchronous Internet 
connection. Alternately, an audio/video stream may contain information indicating that a 
data transfer may be utilized with the audio/video stream. 
30 Figure 1 is a system overview block diagram. Data server 100 employs call 

management system 102 to transfer data across network 104. Call management system 
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102 comprises isochronous data transfer management such as may also be utilized for 
voice data transfers. Call management system 102 employs any protocol that supports 
isochronous data transfers and is not restricted to any particular network or data format. 
Call management system 102 may comprise components from Cisco CallManager 
5 (CCM) software product available from Cicso Systems, a company located at 170 West 
Tasman Dr., San Jose, CA 95134 USA. Network 104 is connected to cable modem 
termination system 106 and to data server 100. Network 104 supports bi-directional data 
transfers and may comprise fiber optic cable, coaxial cable, or other media. Cable modem 
termination system 106 may be implemented using a BSR 64000 broadband services 

1 0 router from RiverDelta Networks that has corporate offices located at 3 Highwood Drive 
Tewksbury, MA 01 876. The aforementioned unit supports DOCSIS and data rate 
controlled transfers such as PacketCable. The cable modem termination system provides 
a quality of service (QoS) level wherein isochronous data transfer is supported. Cable 
modem termination system 106 may support many simultaneous cable modem users on a 

15 single TV channel. If support for more cable modems is desired, additional channels may 
be added to cable modem termination system 106. 

Cable modem 108 communicates with cable modem termination system 106 and 
with host processor 110. Cable modem 108 is a modem (modulator/demodulator) that 
communicates across cable television cable. Cable modem 108 may be an internal or 

20 external cable modem, as may be employed with personal computers, or may be part of 
an interactive set top box. An internal cable modem, or a cable modem that is part of an 
interactive set top box, communicates data through the system's internal bus. An external 
cable modem may employ Ethernet or USB (Universal Serial Bus) to transfer data 
between the modem and the system. 

25 Synchronized service unit 122 comprises demodulator demultiplexer 118, host 

processor 1 10 and A/V decoder 120. Synchronized service unit 122 maybe part of a set 
top box, satellite receiver or personal computer. Host processor 110 may comprise a 
CPU, program memory and system memory. Host processor 1 10 is operable to send and 
receive data to and from data server 100, to process data in system memory, and to output 

30 processed data to A/V decoder 120. Demodulator demultiplexer unit 1 1 8 may be typical 
of that found in a cable set top box receiver and may comprise an A/D converter, QAM- 
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64/256 demodulator, MPEG frame synchronization, and Reed Solomon error correction 
for receive operations. Demodulator demultiplexer unit 1 18 serves as a tuner, receiving 
information contained within a frequency range. Information supplied by demodulator 
demultiplexer unit 1 1 8 to host processor 1 10 and AN decoder 120 may comprise MPEG 

5 data, control or other information. Host processor 1 1 0 may process information from 
demodulator demultiplexer 118, possibly employing data received from data server 100, 
and supply processed information to AN decoder 120, or, host processor 1 10 may allow 
information from demodulator demultiplexer 1 18 to pass directly to AN decoder 120. 
As shown in figure 1, video server 1 12 transfers data through network 1 14 to 

1 0 node 1 16. Video server 1 12 may comprise a server and a cable headend unit such as the 
uBR7200 from Cisco Systems. Network 1 14 may comprise a fiber optic network. Fiber 
optic cable may be employed to carry a signal from a cable television operator's studio or 
production facilities to nodes located in or near viewer neighborhoods. Node 116 may 
provide signal amplification and signal distribution to demodulator demultiplexer unit 

15 118. Node 1 1 6 is an addressable unit in a network and may comprise a computer, 

workstation, or a communications control unit. Video server 112, network 114 and node 
1 1 6 are publicly available components known to cable television broadcast system 
experts. The connection between node 1 1 6 and demodulator demultiplexer 1 1 8 may be 
coaxial cable. 

20 Demodulator demultiplexer unit 1 1 8 supplies signals to host processor 1 1 0 and to 

AN decoder 120. As noted previously, host processor 110 may process information from 
demodulator demultiplexer 1 18 and data from cable modem 108 to produce processed 
information that is sent to AN decoder 120. Processing by host processor 1 1 0 may 
include combining audio and video information and scaling of images. For example, host 

25 processor 1 1 0 may vertically compress a video image to produce a banner area at the 
bottom of the screen, and then display information from the cable modem 108 in the 
banner area. Further, host processor may be employed to parse information from 
demodulator demultiplexer 118 and selectively replace data with data from cable modem 
108. For example, audio information received by demodulator demultiplexer 118 may be 

30 replaced with audio information received by cable modem 1 08, thereby allowing output 
of a foreign language, or language corresponding to a selected audience rating. The 
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information transferred from host processor 1 10 to A/V decoder 120 maybe MPEG data, 
bitmap data, or data in other formats. 

A/V 120 decoder may support analog or digital signal inputs, or both. Output 
from A/V decoder 120 maybe in formats such as composite video, S-video, RGB, and 
5 component audio. Output from A/V decoder 120 may be supplied to display unit 124 
which may comprise a television, computer monitor or other display device. 
Alternatively, output from A/V decoder 120 maybe supplied to a recording device such 
as a video tape recorder, for example. In an alternate embodiment, A/V decoder 120 may 
include processing functions such as overlay and windowing, for example, such that the 

10 images and audio provided to display unit 124 are processed in part by host processor 
110 and may be processed in part by A/V decoder 120. A/V decoder may comprise 
MPEG decoding circuitry, a frame buffer, and circuitry supporting other functions such 
as windowing, menu display, captioning and other functions, for example. 

Figure 2 depicts a flowchart for synchronizing data transfers with an audio/video 

15 stream. At step 200, information is received that identifies an audio/video stream for 
which a data transfer will be available. This information may be contained within an 
audio/video stream or may be provided by another channel or Internet connection. The 
information may contain a descriptor to identify the audio/video stream and a time at 
which a data transfer may be combined with the audio/video stream. Step 202 waits until 

20 the display time of the audio/video stream is within a predetermined period, N, of the 

time at which the data transfer may be combined. In a digital video system, step 202 may 
employ checking of PCR (Program Clock Reference) or PTS (Program Time Stamp), 
queue points or other information within an MPEG stream. In an analog system, step 202 
may employ checking of information contained within the vertical blanking interval to 

25 determine the display time at which data may be combined. Further, a count may be 

established from information contained within the blanking interval and the count may be 
incremented for each vertical blanking interval, thereby establishing an indicator of the 
display time of the presentation, such that step 202 may wait until the indicator of display 
time is within a predetermined period N, of the time at which a data transfer may be 

30 combined with the audio/video stream. Maintaining an indicator of the display time of a 
presentation allows for situations where frame time information may be provided in some 
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but not all blanking intervals. Step 204 checks if a network connection has been 
established. If the result of step 204 is that a network connection is not established, step 
206 establishes a network connection. Step 208 checks to see if data to be combined with 
the audio/video stream has been received. If step 208 determines that data has not been 
5 received, step 210 requests data. The predetermined period, N, of step 202 may be 
established such that there is sufficient time to establish a network connection, request 
and receive data, plus to perform processing of the data. Step 212 processes data 
transfers. Processing may include the rendering of fonts, expansion of compressed data or 
other operations. Step 214 waits until the display time of the audio/video stream is within 

10 a specified period, M, of the time at which the data transfer may be combined with the 
audio/video stream. M may be selected to allow for pipeline processing duration or buffer 
access and processing times. Step 216 combines the processed data with the audio/video 
stream and outputs the result to a television, personal computer or other device. Data may 
include control function information that a set top box, satellite receiver, or personal 

1 5 computer is responsive to. Control function information may alter the output of an audio/ 
video stream such as displaying the video in a smaller window, controlling audio or video 
effects, or other functions. 

Figure 3 depicts a flowchart for personalizing data transfers utilized with an 
audio/video stream. At step 300, the system receives identification information that may 

20 be used to identify advertising or other audio/video streams that may be personalized. 
Step 302 checks if an audio/video stream corresponds to a stream identified by 
information transferred in step 300. When an audio/video stream corresponds to a stream 
identified by information transferred n step 300, step 304 checks if a network connection 
has been established. If a network connection has not been established, step 306 

25 establishes a network connection. Step 308 requests stream information. A server system 
(not depicted) may utilize stream request information that identifies the requestor, such as 
Internet address, cable system identification information, or other information to access 
profile information of the user or a group to which the user may be associated with. 
Stream information is then sent to the client system wherein the information may be 

30 tailored to a characteristic or characteristics of the user. Step 310 receives the stream 
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information. The stream information is employed to obtain data transfers and synchronize 
utilization of data transfers with an audio/video screen as depicted in figure 2. 

Figures 2 and 3 depict steps wherein a check is performed to determine if a 
network connection has been established. In some embodiments of the invention, the 

5 system may employ an 'always connected' architecture such that the aforementioned step 
of determining if a network connection has been established would not be included as 
part of the steps of figures 2 and 3. Further, there may be software that attempts to 
retrieve information a predetermined number of times and after the predetermined 
number of attempts have been performed, the system may discontinue steps to retrieve 

1 0 information and return to the start of the sequence. 

The flowcharts of figures 2 and 3 depict single task threads for simplicity. In 
actual implementation, multitasking may be employed such that during steps that indicate 
waiting for a condition, other processes may be performed. Processes shown in the 
figures may be implemented in hardware, software, or a combination thereof. 

1 5 The foregoing description of the invention has been presented for purposes of 

illustration and description. It is not intended to be exhaustive or to limit the invention to 
the precise form disclosed, and other modifications and variations may be possible in 
light in the above teachings. The embodiment was chosen and described in order to best 
explain the principles of the invention and its practical application to thereby enable 

20 others skilled in the art to best utilize the invention in various embodiments and various 
modifications as are suited to the particular use contemplated. It is intended that the 
appended claims be construed to include other alternative embodiments of the invention 
except insofar as limited by the prior art. 

25 
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